Preregistration

(of secondary data analysis). Why and how.

Jürgen Schneider, Verena Jörg

09 May 2023

Why preregister studies?

Why preregister studies?

Why preregister studies?

It makes Karl Popper happy

  • Critical Rationalism and Falsificationism: Basic orientations in modern philosophy of science


  • we make a prediction (“speculation”) about the future as accurately as possible (rational part)


  • the main activity of a researcher is to invalidate a theory by observation or experiment (critical part)



deductive reasoning / hypothesis testing
(K. Popper, 1935; K. R. Popper, 1989)

Why preregister studies?

It makes Karl Popper happy

does not follow this logic:

  • looking at the world and discover patterns (exploratory research)
    generating vs. testing hypothesis

  • heavily criticized by Popper
    (you will always find support for any claim, so this doesn’t make a strong theory)


e.g. taking a big data set and correlating many variables
(K. Popper, 1935; K. R. Popper, 1989)

Why preregister studies?

Because humans are  idiots  fallable



Does preregistration make any difference at all? In the end, the test is the same!

Why preregister studies?



Kaplan & Irving, 2015

  • NHLBI supported RCTs

  • on treatment or prevention of cardiovascular disease

  • relative risk of primary outcome

  • natural experiment:
    had to be preregistered after year 2000

(Kaplan & Irvin, 2015)

Why preregister studies?



Kaplan & Irving, 2015

  • NHLBI supported RCTs

  • on treatment or prevention of cardiovascular disease

  • relative risk of primary outcome

  • natural experiment:
    had to be preregistered after year 2000

(Kaplan & Irvin, 2015)

Why preregister studies?

Because humans are  idiots  fallable

design: John Manoogian IIIcategories and descriptions: Buster Bensonimplementation: TilmannR, CC BY-SA 4.0, via Wikimedia Commons

Why preregister studies?

Because humans are  idiots  fallable

  • we believe in our theory and want to “prove” it ( critical rationalism!)
  • e.g. confirmation bias
  • questionable research practice (QRP): p-hacking
    • outcome switching
    • unplanned selective reporting
    • unplanned subgroup analyses
    • unplanned sequential testing
    • unplanned covariates
    • unplanned deletion of cases (e.g. “outliers”)
    • …

(John et al., 2012; Munafò et al., 2017)

Why preregister studies?

Because humans are  idiots  fallable

  • we believe that we “knew all along” that the patterns will show up like this ( critical rationalism!)
  • e.g. hindsight bias
  • QRP: HARKING

(Munafò et al., 2017)

Why preregister studies?

1.             



2. Great tool to help us protect ourselves from biased inferences and help others recognize that bias has been tried to be avoided.

BTW: registered reports even more so

Common objections

and objections to these objections
and one affirmation of an objection

Common objections



objection comment
it’s a lot of work it’s mainly front-loading
linking the pre-registration when submitting a paper, uncovers the blinding there are possibilities to blind the prereg on OSF and aspredicted.org
secondary data analysis comes after the data collection?? yes, but you didn’t see the data yet (provide proof of data access date)
yes, because you might already infer relations in data from other known publications

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time. PloS One, 10(8), 0132382. https://doi.org/10.1371/journal.pone.0132382
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Du Percie Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021–0021. https://doi.org/10.1038/s41562-016-0021
Popper, K. (1935). Logik der Forschung: Zur Erkenntnistheorie der Modernen Naturwissenschaft. Springer Vienna.
Popper, K. R. (1989). Logik der Forschung (9., verb. Aufl., Vol. 4). Mohr.

Credit

Icons by Font Awesome CC-BY 4.0

footer